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DETAILED ACTION 
Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1-3, 11, 16-17, 28-29, 32, 37, 42-43, 46, 51 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Schultz et al., "Subpixel Motion Estimation for Super- 
Resolution Image Sequence Enhancement" published by the Journal of Visual 
Communication and Image Representation, v 9 n 1, pages 38-50, March 1998 
("Schultz") in view of U.S. Patent Number 7,088,773 issued to Paniconi et al. 
("Paniconi"). 

For claims 1, 28 and 42, Schultz discloses computing a respective motion map 
for each pairing of a reference image and a respective image neighboring the reference 
image in a sequence of base images, each motion map comprising a set of motion 
vectors mapping reference image pixels to respective neighboring image pixels (page 
38, motion vectors are estimated between video frames, where the accuracy of the 
estimated motion fields has a direct influence on the quality of the high resolution video 
still image). 

Schultz discloses the target image having a target resolution level and the base 
images having a base resolution level equal to or lower than the target resolution level 
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(abstract, a high resolution video still image is estimated from several low resolution 
frames). 

Schultz discloses assigning respective regions of a target image to motion 
vectors based on the computed motion maps, in section 3 entitled "SUBPIXEL MOTION 
ESTIMATION TECHNIQUES," which describes several methods of computing subpixel 
motion vectors for low resolution image pixels. For example, section 3.2 describes a 
block matching motion estimation process that creates a motion map for objects that 
move independently in image sequences, and assigns respective regions of the target 
image to motion vectors based on the computed motion maps by determining separate 
motion fields (or "motion classes") for compact blocks (or "respective regions") in the 
image sequences. 

Schultz further discloses, in section 2 for example, entitled the Bayesian 
multiframe resolution enhancement, computing pixel values for the target image based 
on corresponding pixel value contributions from the base images selected in 
accordance with the motion fields assigned to the target image regions. In other words, 
the super resolution images that are created by Schultz's method are created by pixel 
value contributions from other images selected in accordance with the motion estimates 
assigned to the target image regions as discussed at length by Schultz throughout 
sections 2 and 3. 

To the extent that each motion vector represents a motion class assigned to a 
respective region, and to the extent that each motion field of a compact block is a 
motion class assigned to a respective region, Schultz implicitly describes the concept of 
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motion classes when discussing motion estimation, but Schultz does not explicitly use 
the phrase "motion classes." 

However, performing motion estimation using motion classes is well known in the 
art, as taught by Paniconi, who discloses assigning respective regions of a target image 
to motion classes based on the computed motion maps (figure 2 block 202 computes a 
motion map and block 204 assigns regions to motion classes) and computing pixel 
values for the target image based on corresponding pixel value contributions from the 
base images selected in accordance with the motion classes assigned to the target 
image regions (column 2 lines 39-49, methods for motion segmentation can divide a 
frame into a number of motion classes, where each moving object is assigned to its own 
motion class. See also figure 2). The purpose of performing motion estimation with 
motion classes is to compute pixel values for the target image based on corresponding 
pixel values (or "pixel value contributions") from other (or "base") images selected in 
accordance with the motion classes assigned to the target image regions, as taught by 
Paniconi in col. 1 lines 28-42. 

The use of motion estimation to perform super resolution by "computing pixel 
values for the target image based on corresponding pixel value contributions from the 
base images selected in accordance with the motion [fields] assigned to the target 
image regions" is disclosed by Schultz as discussed above. An example of a motion 
estimation method used by Schultz's super resolution process is Mussman et al., 
"Advances in picture coding," Proceedings of the IEEE 73(4), 1985, 523-548 as 
discussed by Schultz on page 42. Schultz successfully applies Mussman's picture 
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coding method to a super resolution process. Similarly, although Paniconi discloses a 
motion estimation method that was originally designed for a video coding process, it 
would have been obvious to one of ordinary skill in the art to use the motion estimation 
that includes the motion classes of Paniconi when performing the motion estimation in 
the super resolution enhancement algorithm of Schultz, because each motion class can 
be tracked across frames using vectors, which saves processing time as taught by 
Paniconi col. 2 lines 45-49. 

The Supreme Court has held that in analyzing the obviousness of combining 
elements, a court need not find specific teachings, but rather may consider "the 
background knowledge possessed by a person having ordinary skill in the art" and "the 
inferences and creative steps that a person of ordinary skill in the art would employ. 
See KSR Int'l v. Teleflex, Inc., 127 S. Ct. 1727, 1740-41, 82 USPQ2d 1385, 1396 
(2007). To be nonobvious, an improvement must be "more than the predictable use of 
prior art elements according to their established functions." Id. Here the combination is 
the predictable use of two known methods, one performed by the other, according to 
their established functions, to achieve their predictable results. 

Specifically, Schultz discloses the known method of "computing pixel values for 
the target image based on corresponding pixel value contributions from the base 
images selected in accordance with the motion [estimates] assigned to the target image 
regions" and Paniconi discloses the known method of creating "motion classes 
assigned to the target image regions" when performing motion estimation. Therefore, 
the combination of "computing pixel values for the target image based on corresponding 
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pixel value contributions from the base images selected in accordance with the motion 
classes assigned to the target image regions" yields a predictable result of using motion 
classes to improve the motion estimation part of a super resolution method. 

For claim 2, Schultz discloses generating for each image pair respective dense 
motion vectors describing motion at pixel locations with respective sets of parameters in 
a motion parameter space at sub-pixel accuracy (the entire article starting with the title 
and ending with the conclusion discusses this claimed element). 

For claims 3, 29 and 43, Paniconi discloses assigning pixels of the reference 
image to respective motion classes and up-projecting the motion class assignments to 
pixels of the target image (see for example figures 1a, 1b and 1c which show classes 
projected to pixels, see also column 6 lines 1-24). 

For claims 1 1 , 32 and 46, Schultz discloses computing an alignment accuracy 
map for each pairing of the reference image and a respective neighboring image based 
on the computed motion maps (page 43, the displaced frame difference along with its 
mean and variance are computed to determine how well the motion vectors have been 
estimated). 
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For claim 16, Schultz discloses up-projecting the motion maps from the base 
image resolution level to the target image resolution level (see page 45, the up-sampled 
frames produce the up-sampled subpixel resolution motion vectors). 

For claims 17, 37 and 51, Schultz discloses re-mapping the neighboring images 
to the reference frame of the up-projected reference image using the respective up- 
projected motion maps (page 45, re-mapping is performed by computing the Bayesian 
multiframe HRVS for each motion field). 

Claims 4-5, 30 and 44 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Schultz and Paniconi as applied to claims 1-3 above, and further in 
view of U.S. Patent Number 6,269,175 issued to Hanna et al. ("Hanna"). 

For claims 4, 30 and 44, Paniconi discloses computing motion magnitude maps 
from each motion map (see columns 8-9 and figures 6a, 6b, 6c, 6d, 6e and 6f, although 
Paniconi does not use the words "motion magnitude maps," he clearly computes the 
motion magnitude map for each class from the motion map as shown for example in the 
discussion of figure 6c), Hanna discloses multiresolution image pyramid representations 
that can down-sample the computed motion magnitude maps of Paniconi to a pyramid 
of motion magnitude maps at respective resolution levels lower than the base resolution 
level, and segment pixels in the pyramid of down-sampled motion magnitude maps into 
motion classes. 
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It would have been obvious to one of ordinary skill in the art at the time of 
invention to include the multiresolution image pyramid resolutions with the image 
enhancement and segmenting methods of Schultz and Paniconi, because the pyramid 
resolutions provide the benefit of very efficient computation even when large 
displacements are present but also provide subpixel accuracy in displacement 
estimates as taught by Hanna at column 6 lines 40-50. 

For claim 5, Paniconi discloses segmenting, and Hanna discloses iteratively 
estimating pixels in the pyramid of motion magnitude maps from a coarsest resolution 
level up to the base resolution level, wherein estimating from each coarser resolution 
level is up-sampled to initialize the estimation at a finer resolution level, and refined by 
further estimation at the finer resolution level. 

Claims 6, 12-15, 31, 33-36, 45, 47-50 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Schultz and Paniconi as applied to claims 1-3 above, and 
further in view of Eren et al., "Robust, Object-Based High-Resolution Image 
Reconstruction from Low-Resolution Video," published by IEEE Transactions on Image 
Processing, v 6 n 10, October 1997 ("Eren"). 

For claims 6, 31 and 45, Eren discloses generating a separate motion class 
segmentation map for each pairing of the reference image and a respective neighboring 
image and merging the separate motion class segmentation maps into a unified motion 
class segmentation map for the reference image (see pages 1448-49 and figures 1(a) 
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and (e), the segmentation maps of each frame are given or can be computed, and the 
segmentation map of a reference frame is given. The maps are used to create a high 
resolution mosaic). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify the enhancement methods of Schultz and Paniconi with the 
segmentation map of Eren to enable object based processing that uses more accurate 
motion models and improves the quality of the reconstructed image as taught by Eren in 
the abstract. 

For claims 12, 33 and 47, Eren discloses re-mapping neighboring images to a 
coordinate frame of the reference image using respective motion maps, and computing 
correlation measures between pixels of the reference image and pixels of each of the 
motion-compensated neighboring images (see section III robust high-resolution 
reconstruction using a validity map, the correlation measure is determined over a 3x3 
window of pixels between the reference image and other low resolution images). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to modify the enhancement methods of Schultz and Paniconi with the validity 
map of Eren to improve the quality of the image by using projections only for those pixel 
locations for which the motion vectors are accurate as taught by Eren. 
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For claims 13, 34 and 48, Eren discloses up-projecting the computed alignment 
accuracy maps from the base image resolution level to the target image resolution level 
as discussed in section III. 

For claims 14, 35 and 49, Eren discloses up-projecting the motion maps from the 
base image resolution level to the target image resolution level, and classifying motion 
vectors in each up-projected motion map into valid and invalid motion vector classes 
based on the up-projected alignment accuracy maps as disclosed throughout the entire 
document's discussion of the validity map. 

For claims 15, 36 and 50, both Eren and Schultz disclose values of target image 
pixels with corresponding pixels in all neighboring images being associated with motion 
vectors in the invalid motion vector class are computed by interpolating up-projected 
pixel values of the reference image. 

Claims 7-10, 18-27, 38-41, 52-55 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Schultz, Paniconi and Eren as applied to claim 6 above, and further 
in view of U.S. Patent Numer 6,307,560 issued to Kondo et al. ("Kondo"). 

For claim 7, Paniconi discloses that reference image pixels are respectively 
assigned to a motion class. Kondo discloses that the motion class is selected from a 
motion class set including a high motion class and a low motion class, and motion 
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vectors assigned to the high motion class have higher magnitudes than motion vectors 
assigned to the low motion class (column 5 lines 18-36). 

It would have been obvious to one of ordinary skill in the art at the time of 
invention to include the low and high motion classes of Kondo with the motion classes 
of Paniconi for the benefit of varying the description of the motion according to the 
application as taught by Kondo in column 5 line 25. 

For claim 8, Paniconi discloses assigning a given reference image pixel to the 
low motion class in the unified motion class segmentation map when the given pixel is 
assigned to the low motion class in all of the separate motion class segmentation maps, 
and assigning a given reference image pixel to the high motion class in the unified 
motion class segmentation map when the given pixel is assigned to the high motion 
class in any of the separate motion class segmentation maps (see figures 1a, 1b and 1c 
each object has its own class that is tracked over multiple frames). 

For claim 9, Kondo discloses that the motion class set further includes an 
intermediate motion class, and motion vectors assigned to the intermediate motion 
class have magnitudes higher than motion vectors assigned to the low motion class and 
lower than motion vectors assigned to the high motion class (see column 5). 
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For claim 10, Kondo discloses assigning a given reference image pixel to the 
intermediate motion class (column 5) and Paniconi and Eren disclose the unified motion 
class segmentation map. 

For claim 18, Paniconi discloses regions of the target image are respectively 
assigned to a motion class selected from a motion class set (abstract) and Kondo 
discloses the motion classes include a high motion class and a low motion class, and 
motion vectors of regions assigned to the high motion class have higher magnitudes 
than motion vectors of regions assigned to the low motion class (column 5) 

For claims 19, 38 and 52, Eren suggests computing a pixel-wise combination of 
pixel value contributions from the up-projected reference image and the re-mapped 
neighboring images weighted based on pixel-wise measures of alignment accuracy 
between the reference image and the corresponding neighboring images (the validity 
map and blurring is accounted for to determine whether pixels are aligned and 
contribute). 

For claims 20, 39 and 53, Eren suggests that pixel value contributions from the 
re-mapped neighboring images are additionally weighted based on measures of 
temporal distance between the reference image and the corresponding neighboring 
images (the temporal distance is accounted for using motion vectors and the 
segmentation maps). 
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For claims 21, 40 and 54 Eren suggests classifying low motion class reference 
image pixels and their corresponding pixels in the neighboring images based on 
measures of local texture richness (see page 1449 the background is low motion that is 
classified based on texture). 

For claim 22, Eren and Paniconi suggest low motion class reference image pixels 
and their corresponding pixels in the neighboring images are quantitatively evaluated for 
local texture richness, and are classified into a texture class selected from the texture 
class set including a high texture region class and a low texture region class, and pixels 
assigned to the high texture region class have higher local texture measures than pixels 
assigned to the low texture region class (see page 1448 of Eren and the abstract of 
Paniconi, the low motion classes can be segmented into low motion background and 
low motion object based on the textures). 

For claim 23, Eren suggests values of target image pixels classified into the low 
texture region class in the reference image and all of the respective neighboring 
images, are computed by interpolating up-projected pixel values of the reference image 
(the pixel values of the background are determined from the reference image). 

For claim 24, Eren and Paniconi suggest a value of a given target image pixel 
classified into the high texture region class in the reference image or any respective 
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neighboring images is computed based on a pixel value contribution from the up- 
projected reference image, and a pixel value contribution from a given re-mapped 
neighboring image weighted based on a measure of local texture richness computed for 
the given pixel, a measure of motion estimation accuracy computed for the given pixel, 
and a measure of temporal distance of the neighboring image from the reference image 
(the pixel values of a moving object are determined from texture, motion, and temporal 
images). 

For claims 25, 41 and 55, Eren discloses values of target image pixels are 
computed based on pixel value contributions from a number of base images 
neighboring the reference image, and Paniconi discloses the number of neighboring 
base images being different for different motion classes (if an object is moving faster 
than another object, it will usually be present in fewer base images before it moves out 
of the field of view). 

For claim 26, the combination of Eren, Paniconi and Kondo suggests values of 
target image pixels assigned to the high motion class are computed based on pixel 
value contributions from a fewer number of neighboring base images than the values of 
target image pixels assigned to the low motion class (slower objects are present in more 
images than faster objects). 



Application/Control Number: 10/824,692 Page 15 

Art Unit: 2624 

For claim 27, Kondo suggests an intermediate motion class, the motion vectors 
of regions assigned to the intermediate motion class have lower magnitudes than 
motion vectors of regions assigned to the high motion class and higher magnitudes than 
motion vectors of regions assigned to the low motion class, and values of target image 
pixels assigned to the intermediate motion class are computed based on pixel value 
contributions from a fewer number of neighboring base images than the values of target 
image pixels assigned to the low motion class but a higher number of neighboring base 
images than the values of target image pixels assigned to the high motion class. 

Response to Arguments 

Applicant's arguments filed October 22 have been fully considered but they are 
not persuasive. 

In response to applicant's argument that "the mere teaching that 'methods for 
motion segmentation can divide a frame into a number of motion classes, where each 
moving object is assigned to its own motion class' does not constitute a teaching of ... 
assigning respective regions of a target image to motion classes based on the 
computed motion maps," this is precisely what Paniconi's method teaches and 
discloses in this section and throughout the entire patent beginning with the title and 
ending with the claims. In particular, figure 2 blocks 202 and 204 show this claim 
element as discussed in the rejection of claim 1 . 
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In response to applicant's arguments that Paniconi does not disclose "computing 
pixel values for the target image based on corresponding pixel value contributions from 
the base images selected in accordance with the motion classes assigned to the target 
image regions" the super resolution portion of this element (computing pixel values for 
the target image based on corresponding pixel value contributions from the base 
images selected in accordance with the motion vectors assigned to the target image 
regions) is disclosed by Schultz and "the motion classes assigned to the target image 
regions" are disclosed by Paniconi. See for example the entire patent beginning with 
the title and ending with the claims. 

In response to applicant's arguments against the references individually, one 
cannot show nonobviousness by attacking references individually where the rejections 
are based on combinations of references. See In re Keller, 642 F.2d 413, 208 
USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 
1986). In this case, Schultz discloses "computing pixel values for the target image 
based on corresponding pixel value contributions from the base images selected in 
accordance with the motion [estimation] assigned to regions of the target image" and 
Paniconi discloses motion estimation that includes "motion classes assigned to the 
target image regions." Therefore, the elements of the combination "computing pixel 
values for the target image based on corresponding pixel value contributions from the 
base images selected in accordance with the motion classes assigned to the target 
image regions" are disclosed by the prior art. 
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However, if applicant desires to argue each reference separately, each reference 
by itself anticipates claim 1 . 

Paniconi by itself anticipates claim 1 . To the extent that the base images have a 
resolution "equal to" the "target resolution level," the claim is not limited to super 
resolution but rather includes the standard compression method disclosed by Paniconi. 
That is, Paniconi discloses "computing pixel values for the target image based on 
corresponding pixel value contributions from the base images selected in accordance 
with the motion classes assigned to the target image regions" where "the base images 
have a resolution equal to ... the target resolution level." See for example columns 1 
and 2 which discuss using motion classes to generate pixel values for a target image 
based on corresponding pixel value contributions from base images selected in 
accordance with the motion classes, where the base images have a resolution equal to 
the target resolution level. 

Schultz by itself anticipates claim 1. Schultz discloses computing a respective 
motion map for each pairing of a reference image and a respective image neighboring 
the reference image in a sequence of base images, each motion map comprising a set 
of motion vectors mapping reference image pixels to respective neighboring image 
pixels (page 38, motion vectors are estimated between video frames, where the 
accuracy of the estimated motion fields has a direct influence on the quality of the high 
resolution video still image). Schultz discloses the target image having a target 
resolution level and the base images having a base resolution level equal to or lower 
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than the target resolution level (abstract, a high resolution video still image is estimated 
from several low resolution frames). Schultz discloses assigning respective regions of a 
target image to motion classes based on the computed motion maps in section 3.2, 
which describes a block matching motion estimation process that creates a motion map 
for objects that move independently in image sequences, and assigns respective 
regions of the target image to motion classes based on the computed motion maps by 
determining separate motion fields (motion classes) for compact blocks (respective 
regions of the target image) in the image sequences. Schultz further discloses, in 
section 2, computing pixel values for the target image based on corresponding pixel 
value contributions from the base images selected in accordance with the motion 
classes assigned to the target image regions. In other words, the super resolution 
images that are created by Schultz^ method are created by pixel value contributions 
from other images selected in accordance with the motion classes (such as motion 
fields and motion vectors) assigned to the target image regions (such as compact 
blocks) as discussed at length by Schultz throughout section 3. 

Nevertheless, even though Paniconi and Schultz each individually anticipate 
claim 1 , in order to advance prosecution the superresolution element of "a base 
resolution level ... lower than the target resolution level" and the element of "computing 
pixel values for the target image based on corresponding pixel value contributions from 
the base images selected in accordance with the motion classes assigned to the target 
image regions" have been addressed by the combination of references. 
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In response to applicant's argument that there is no suggestion to combine the 
references, The Supreme Court has held that in analyzing the obviousness of 
combining elements, a court need not find specific teachings, but rather may consider 
"the background knowledge possessed by a person having ordinary skill in the art" and 
"the inferences and creative steps that a person of ordinary skill in the art would employ. 
See KSR Int'l v. Teleflex, Inc., 127 S. Ct. 1727, 1740-41, 82 USPQ2d 1385, 1396 
(2007). To be nonobvious, an improvement must be "more than the predictable use of 
prior art elements according to their established functions." Id. Here the combination is 
the predictable use of two known methods, one performed by the other, according to 
their established functions, to achieve their predictable results. In this case, Schultz 
discloses the known method of "computing pixel values for the target image based on 
corresponding pixel value contributions from the base images selected in accordance 
with motion [estimates] assigned to the target image regions" and Paniconi discloses 
the known method of motion estimation that creates "motion classes assigned to the 
target image regions." Therefore, the combination of "computing pixel values for the 
target image based on corresponding pixel value contributions from the base images 
selected in accordance with the motion classes assigned to the target image regions" 
yields a predictable result. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jeffrey S. Smith whose telephone number is 571 270- 
1235. The examiner can normally be reached on M-F. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on 571 272-7429. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

JSS 

December 18, 2007 



